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We search for the production of a heavy W' gauge boson that decays to third generation quarks in 
0.9 fb _1 of pp collisions at y/s = 1.96 TeV, collected with the DO detector at the Fermilab Tevatron 
collider. We find no significant excess in the final-state invariant mass distribution and set upper 
limits on the production cross section times branching fraction. For a left-handed W' boson with 
SM couplings, we set a lower mass limit of 731 GeV. For right-handed W' bosons, we set lower mass 
limits of 739 GeV if the W 1 boson decays to both leptons and quarks and 768 GeV if the W' boson 
decays only to quarks. We also set limits on the coupling of the W' boson to fermions as a function 
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of its mass. 

PACS numbers: 13.85.Rm; 14.65.Ha; 14.70.Pw; 12.15.Ji; 13.85 Qk 



New massive charged gauge bosons, usually called 
W, are predicted by various extensions of the standard 
model (SM). Noncommuting extended technicolor, little 
Higgs, composite gauge bosons, grand unification, and 
superstring theories represent examples in which an 
extension of the gauge group leads to the appearance 
of a W boson fl. 

Direct searches for such W' bosons in leptonic final 
states {iv) lead to the lower limit M w > > 1.0 TeV @, 
assuming the W boson couples to fermions in the same 
way as the SM W boson. W' bosons that couple to 
right-handed fermions may not be able to decay to 
leptonic final states if the corresponding right-handed 
neutrinos are too massive. In this case, only decays to 
qq' final states are possible, and the best limit, based 
on decays of the W boson to two light quark jets is 
Mw > 800 GeV [§]. There are model-dependent upper 
limits on the mass of W bosons, based on cosmological 
and astrophysical data, that range from 549 GeV to 
23 TeV Q. 

In this Letter, we report a search for a W boson that 
decays to third generation quarks (W — > tb or tb). For 
brevity, we will use the notation tb to represent the sum 
of the tb and the tb decay modes. 

A W' boson that decays to tb contributes to single 
top quark production [f| for which evidence has been 
reported recently Since the SM W boson and a 

hypothetical W' boson with left-handed couplings both 
couple to the same fermion multiplets, they interfere with 
each other. The interference term may reduce the total 
rate by as much as (16-33)%, depending on the mass of 
the W boson and its couplings [l(. Previous searches Q 
in this channel (neglecting interference effects) at the 
Tevatron have led to the 95% C.L. limits M w , > 536 GeV 
if the W decays to Iv and to qq' and Mw > 566 GeV if it 
only decays to qq' . A recent DO analysis Q, which takes 
into account the interference, excludes masses between 
200 GeV and 610 GeV for a W boson with left-handed 
SM-like couplings, between 200 GeV and 630 GeV for 
a W' boson with right-handed couplings that decays to 
Iv and qq' , and between 200 GeV and 670 GeV for 
a W boson with right-handed couplings that can only 
decay to qq'. 

The most general lowest-order effective Lagrangian for 
the interactions of a W boson with SM fermions / with 
generation indices i and j, is 

where Vij is the Cabibbo-Kobayashi-Maskawa [9j matrix 
element if the fermion is a quark, and Vy = if it 
is a lepton, 5ij is the Kronecker delta, g w is the weak 



coupling constant of the SM, and a^ , a^j are coefficients. 
In this notation, aji- = 1 and a$ = for a so-called 
SM-like W' boson. This effective Lagrangian has been 
incorporated into the COMPHEP pa ckag e 1 101 ] and used 
by the SINGLETOP event generator [111 ]. SINGLETOP is 
used to simulate SM single top quark production via the 
exchange of a W boson in the s- and i-channel, and the 
s-channel W signal, including interference with the SM 
W boson. We simulate the complete chain of W, top 
quark, and W boson decays, taking into account finite 
widths and all spin correlations between the production 
of resonance states and their decay. The top quark mass 
is set to 175 GeV, the CTEQ6L1 parton distribution 
functions [12] are used and the factorization scale is set 
to Mw Next-to-leading-order (NLO) corrections are 
included in the SINGLETOP generator, and normalization 
and matching between various partonic subprocesses are 
performed such that not only the rates, but also the 
shapes of distributions at NLO [l3f . are reproduced. 

We generate samples of purely left-handed W' L bosons 
1 and afj — 0, and purely right-handed 
= and = 1. W' T bosons interfere 
with the standard W boson, but W' R bosons couple 
to different final state particles and therefore do not 
interfere with the standard W boson. The iv decays of 
W R bosons involve a right-handed neutrino of unknown 



with afj 

W R bosons with af^ 



mass, assumed to be M VR > Mw 



or M VR < Mw 



The W width varies between 20 GeV and 30 GeV for 
W masses between 600_GeV and 900 GeV 0, (H. If 
M VR > Mw and only qq' final states are open, the width 
is about 25% smaller. This does not have a significant 
effect on our search as the experimental resolution for 
the tb invariant mass is much larger (ss 90 GeV). The 
branching fraction for W' — * tb, is around 0.32 (0.24) 
for decays only to quarks (quarks and leptons) for a 
W boson with a mass of 700 GeV and varies slightly 
with the mass. In the absence of interference between 
W and W' bosons, and if M VR < M\y>, there is no 
difference between W' L and W' R for our search. Since 
the current lower limit on the mass of the W boson is 
around 600 GeV we simulate W' L and W' R bosons at 
seven mass values from 600 to 900 GeV to probe for W' 
bosons with higher masses. 

We analyze events with leptons, jets, and missing 
transverse momentum, pr, in the final state. The data 



were recorded by the DO detector |14( between 2002 and 
2005 using triggers that required a jet and an electron 
or a muon. They correspond to 0.9 fb _1 of integrated 
luminosity. The event selection and trigger criteria are 
very similar to those in Ref. 0] and require exactly one 
isolated electron (muon) with a momentum component 
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transverse to the beam direction pr > 15 (20) GeV and 
pseudorapidity \r)\ < 1.1 (2.0), 15 < pr < 200 GeV, a 
leading jet with px > 25 GeV and |?7| < 2.5, and a 
second leading jet with p T > 20 GeV and |?7| < 3.4. 
We select events with two or three jets, counting all jets 
with pt > 15 GeV and < 3.4. Events with more 
than three jets are excluded to reduce the it background. 
Jets are reconstructed using the Run II midpoint cone 
algorithm [l5| with cone size 0.5. Since we expect two 
b quarks in the W — ► tb decay, we require at least 
one jet to be classified ("tagged") as a b jet [16]. The 
data are divided into eight independent channels based 
on lepton flavor (e, fi), jet multiplicity (2, 3), and 
number of 6-tagged jets (1, >2) to take into account 
the different signal acceptances and signal-to-background 
ratios, which increases the sensitivity of the search. 

Background yields are estimated using both Monte 
Carlo (MC) samples and data in the same way as in 
Ref. @ . Control data samples are used to determine the 
multijet background from events in which a jet is misiden- 
tified as an electron, or a muon from a semileptonic heavy 
flavor quark decay is considered to be from the decay 
of a W boson. The tt background is estimated using 



:a) D0O.9fb 1 



the ALPGEN [17j MC event generator, normalized to the 
theoretical cross section of 6.8 ±1.2 pb [HI]. The Ty+jets 
background is modeled using ALPGEN and its yield is 
normalized together with the multijet background, so 
that the total background yield equals the observed 
number of events in data before requiring a 6-tagged jet. 
The fraction of VF+jets events with heavy flavors (Wbb, 
Wcc) is measured using data. In this way, the small 
contributions from Z+jets and diboson processes (WW, 
WZ, ZZ) are absorbed into the MK+jets background 
normalization. For the W' R search, the SM single top 
quark production is included in the background. Because 
of their interference, the s-channel single top quark 
production is considered part of the signal for the W' L 
search and only the i-channel single top quark production 
is included in the background. All parton-level MC 
samples are further processed with pythia [l^] and a 
GEANT [201] -based simulation of the DO detector. Lepton 
and jet energies are corrected to reproduce the resolutions 
observed in data. 

The distinguishing feature of a W' signal is a resonance 
structure in the tb invariant mass. However, we cannot 
directly measure the tb invariant mass. Instead we 
reconstruct the invariant mass v» of the leading two 
jets, the charged lepton, and the neutrino by adding 
their measured momentum four- vectors. The missing 
transverse momentum is used to obtain the x and 
y-components of the neutrino momentum and the z- 
component is the smaller of the two \p"\ values that 
makes the iv mass equal the W boson mass. 

The observed VI distribution in the data is consistent 
with the background prediction within uncertainties and 
shows no evidence for a signal (see Fig. [T]). Since we 
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FIG. 1: Vs distributions for the data and the SM background. 
Plot (a) shows the W' L signal and plot (b) shows the W' R signal 
at two different masses, normalized to the NLO prediction 
( Table Hljl . Events in the eight subsamples (electron, muon, 
two and three jets, single-tagged and double-tagged) are 
combined. 



search for W' bosons with masses greater than 600 GeV, 
we set upper limits on the W' boson production cross 
section times branching fraction to the tb final state, 
a(pp -> W') x B(W' -> tb), using the high tail of the 
y/I distribution. Table [J gives the observed number 
of data events and the expected background yields for 
events with y/1 > 400 GeV (chosen to improve the signal 
to background ratio). The signal yields corresponding 
to the same selection are listed in Table ID We use a 
Bayesian method [2~0 ] with a flat nonnegative prior for 
the signal cross section. The limits are derived using 
a binned likelihood constructed from the y/1 spectrum 
above 400 GeV, taking into account all systematic 
uncertainties, and their correlations. We also compute 
expected upper limits as a measure of the sensitivity 
of the analysis. We combine the eight independent 
subsamples to obtain the limits listed in Table |TTJ 
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TABLE I: Data and SM background event yields. 



Process 


Events 




oNl-\-vv £ SGarcn w r searcn 


Single top 


6.4 ± 1.4 10.2 ± 2.2 


ti 


59.1 ± 14.4 


W+jets 


91.0 ± 18.8 


Multijets 


29.7 ± 5.9 


Total backj 


'round 186.1 ± 40.4 190.0 ± 41.2 


Data 


182 



TABLE II: NLO production cross sections x branching 
fraction to tb in pb (theory), expected signal event yields 
(evts), and expected (exp) and observed (obs) 95% C.L. upper 
limits for a(pp -» W') x B(W' -> tb) in pb. Theory I (II) 
corresponds to the case M VR < M w i (M„ H > M w i). The 
uncertainty on signal yields are around 20%. 



(GeV) Theory Evts Exp Obs 



Theory Evts Exp Obs 
(I) (II) 



600 
650 
700 
750 
800 
850 
900 



2.17 
1.43 
1.01 
0.76 
0.62 
0.55 
0.51 



58 
33 
19 
11 
6 
4 
3 



0.69 
0.65 
0.69 
0.80 
1.04 
1.46 
2.35 



0.66 
0.69 
0.74 
0.93 
1.23 
1.77 
2.79 



2.10 
1.25 
0.74 
0.44 
0.26 
0.16 
0.09 



2.79 
1.65 
0.97 
0.57 
0.34 
0.20 
0.12 



61 

35 
20 
12 
7 
4 
2 



0.67 
0.55 
0.50 
0.44 
0.42 
0.42 
0.40 



0.58 
0.59 
0.54 
0.50 
0.47 
0.48 
0.44 



In the evaluation of the systematic uncertainties, 
we take into account the uncertainties in both the 
background normalization and the shape of the 
distribution. Uncertainties in the integrated luminosity 
(6.1%), theoretical cross sections [(15-18)%], branching 
fractions (1%), object identification efficiencies [(1- 
7.5)%], trigger efficiencies [(3-6)%], jet fragmentation 
modeling (5%), and the uncertainty in the fraction of 
W+jets events with heavy quarks affect only the normal- 
ization Uncertainties in the 6-jet simulation [(12- 
17)%] and the jet energy scale calibration [(1-20)%] affect 
both shape and normalization. Ranges represent the 
variations among the eight subsamples. 

The observed 95% C.L. upper limit of a(pp — > W) x 
B(W — > tb) compared to the NLO theory predictions 
are shown in Fig. [3] for (a) W' L and (b) W' R production 
cross sections. For the W' L boson, we show the total 
cross section for s-channel single top quark production 
including the SM diagram, the W diagram, and their 
interference [l[ ■ In this case the limit applies to the total 
s-channel single top production. The /c-factors needed 



to scale the W' L cross section to NLO, the NLO cross 
sections for the W' R boson, and the expected theoretical 
uncertainty are taken from Ref. [13]. Using the nominal 
(nominal — la) values of the theoretical cross section, the 



lower limit for W' L mass is 731 (718) GeV. For W' R bosons 
that decay only to qq the limit is 768 (750) GeV; 739 
(725) GeV if the leptonic decay is also allowed. 

Limits for the gauge couplings g' = g w afj or g' = 
g w aij, depending on the model, of the W boson can 
be derived from the cross section limits. Since the 
leading-order s-channel production diagram has two 
W'qq' vertices, o{pp — > W) x B(W — * tb) is propor- 
tional to g' A . Figure \2[c) shows the observed limit for 
g'/gw We exclude gauge couplings above 0.68 (0.72) g w 
for W' bosons with a mass of 600 GeV for the case 
M VR > M w , (M VR < M w ,). 

We have performed a search for W' bosons that decay 
to tb, using 0.9 fb" 1 of data recorded by the DO detector. 
We find no evidence for W boson production and set 
95% C.L. upper limits on a{pp W) x B(W -> 
tb). We use the nominal value of the theoretical cross 
section to set limits on the mass of the W' bosons. We 
exclude W' L bosons with left-handed, SM-like couplings 
with masses below 731 GeV. For W' R bosons with right- 
handed couplings, we set a lower mass limit of 739 GeV 
if the W boson can decay to leptons and to quarks. If 
the W decays only to quarks, the lower mass limit is 
768 GeV. We also constrain the W gauge coupling and 
exclude couplings above 0.68 (0.72) g w for W bosons 
with a mass of 600 GeV that only decay to quarks 
(leptons and quarks). These limits represent a significant 
improvement over previously published results 0, Q • 
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